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ABSTRACT 

We  provide  a closed-form  solution  for  obtaining  3-D  structure  of  a scene  for  a 
given  six  degree  of  freedom  motion  of  a camera.  The  solution  is  massively  parallel, 
i.e.,  the  range  that  coiresponds  to  each  pixel  is  dependent  on  the  spatial  and  temporal 
changes  in  intensities  of  that  pixel,  and  on  the  motion  parameters  of  the  camera.  The 
measurements  of  the  intensities  are  done  in  a priori  known  directions.  The  solution 
is  for  the  general  case  of  camera  motion. 

The  derivation  is  based  upon  representing  the  image  in  the  spherical  coordinate 
system,  although  a similar  approach  could  be  taken  for  other  image  domains,  e.g.,  the 
planar  coordinate  system.  We  comment  on  the  amount  of  computations,  errors  and 
singular  points  of  the  solutions.  We  also  suggest  a practical  way  to  significantly  reduce 
and  implement  them. 
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1.  INTRODUCTION 

ITie  perception  of  depth  is  critical  to  survival  for  creatures  of  nature.  Depth  per- 
ception is  necessary  for  locomotion  without  collision  in  a complex  natural  3-D  world.  It 
is  also  advantageous  for  recognizing  objects.  In  a natural  environment,  image  segmenta- 
tion bas^  on  brightness  boundaries  of  stationary  objects  is  extremely  difficult.  It  is 
confounded  by  shadows,  surface  texture,  and  ma'kings  that  often  produce  brighmess 
and  color  boundaries  that  are  much  more  distinct  than  true  object  or  surface  boundaries. 
In  the  natural  world,  segmentation  based  on  motion  discontinuities  is  much  more  reli- 
able than  segmentation  based  on  brighmess  discontinuities. 

Many  methods  have  been  proposed  for  computing  depth  from  visual  input.  Of 
these,  only  the  methods  of  stereo  and  image  flow  can  compute  depth  directly  without 
any  Imowledge  of,  or  understanding  about,  the  contents  of  the  image.  Only  image  flow 
can  compute  depth  from  a single  retina  without  simultaneously  viewing  the  scene  from 
two  separate  eyes.  Depth  from  image  flow  is  thus  extremely  important  to  prey  animals, 
such  as  fish,  most  birds,  and  rabbits  because  their  eyes  diQ  on  the  opposite  sides  of 
their  heads.  If  they  have  stereo  vision  at  all,  it  covers  very  little  of  the  visual  field.  Yet 
such  animals  can  swim,  fly,  or  run  rapidly  through  complex  natural  environments  (such 
as  reefs,  forests,  or  thickets)  filled  with  obstacles  without  collisions.  They  can  also 
effectively  flee  predators  and  manuever  relative  to  other  members  in  flocks,  herds,  and 
schools. 

Image  flow  is  motion  of  an  image  on  the  retina  of  the  eye,  or  on  the  photodetector 
amay  of  a camera.  Image  flow  can  be  caused  by  two  phenomena:  1)  motion  of  objects 
in  the  world,  and  2)  motion  of  the  eye  (camera)  through  the  world.  Motion  of  the  eye 
can  be  of  two  types:  translation  and  rotation.  The  algorithm  described  here  applies  to 
image  flow  produced  by  translation  and  rotation  of  the  eye  in  a stationary  world. 

It  is  well  known  that  image  flow  due  to  translation  is  by  itself  a sufficient  visual 
cue  for  safe  movement  through  a natural  environment  filled  with  complex  3-D  obsta- 
cles. Helmholtz  [1]  observed  as  early  as  1925  that  monocular  image  flow  provides 
direct  and  immediate  perception  of  3-D  space.  The  rate  of  apparent  motion  of  a sta- 
tionary object  in  the  world  induced  by  translation  of  the  eye  through  the  world  is 
inversely  proportional  to  the  distance  from  the  eye  to  the  object.  Close  objects  which 
are  potential  collision  hazards  produce  large  image  flow.  Distant  objects  or  close 
objects  near  the  instantaneous  axis  of  motion  produce  little  or  no  image  flow. 

The  psychophysics  of  image  flow  was  extensively  studied  by  Gibson  [2],  who 
treated  the  perception  of  space  as  the  perception  of  a collection  of  surfaces,  and  con- 
sidered motion  relative  to  a surface  to  be  the  most  fundamental  of  visual  perceptions. 
Rogers  and  Graham  [3]  have  shown  that  random-dot  displays  of  monocular  flow  pat- 
terns produce  the  perception  of  solid  oriented  surfaces,  and  that  image  flow  is  adequate 
for  shape  and  depth  perception  with  no  other  depth  information. 

Methods  for  computing  image  flow  from  time-varying  images  have  been  proposed 
by  Horn  and  Schunck  [4],  Longuet-Higgins  and  Prazdny  [5],  Hildreth  [6],  Waxman  and 
Wohn  [7],  Waxman,  Kamgar-Parsi,  and  Subbarao  [8],  and  others.  In  most  of  this 
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work,  camera  motion  is  assumed  to  be  unknown.  Overcoming  this  assumption  requires 
complex  mathematical  analyses  that  are  subject  to  error  from  quantization  noise,  and 
are  too  computationally  intensive  for  practical  real-time  implementation.  Recent  work 
by  Bolles  [9],  Matthies,  Szeliski  and  Kanade  [10],  Albus  and  Hong  [13],  Rangachar, 
Hong,  and  Herman  [16],  and  Heel  [15]  has  been  based  on  knowing  camera  motion. 
Combined  with  Kalman  filtering  techniques  this  approach  has  produced  dense  (iconic 
pixel  based)  range  maps  with  relative  RMS  error  values  of  less  than  three  percent. 
Except  for  Heel,  the  approach  has  been  defined  only  for  translation  without  rotation 
with  the  camera  scan  lines  aligned  with  an  epipolar  plane. 

The  algorithm  given  here  treats  the  general  case  of  simultaneous  camera  transla- 
tion and  rotation.  It  assumes  both  to  be  known.  We  assert  this  to  be  a good  assump- 
tion, because  most  creatures  of  nature  (and  robot  vehicles  as  well)  possess  non-visual 
vestibular  and  proprioceptive  sensors,  and  other  sensory-motor  systems  that  enable 
them  to  estimate  three-axis  eye  rotation,  as  well  as  ground  speed,  air  speed,  or  water 
speed  independently  of  visual  input.  This  estimate  can  be  refined  by  the  addition  of 
global  visual  data  integrated  over  the  entire  visual  field.  This  paper  thus  assumes  that 
there  exists,  either  in  the  brain  or  the  robot  system,  a mechanism  by  which  both  visual 
and  non-visual  information  is  fused  into  a single  best  estimate  of  eye  translation  and 
rotation. 

This  paper  provides  a closed  form  solution  to  the  computation  of  range  to  every 
pixel,  and  suggests  a massively-parallel  computational  structure  that  could  make  range 
from  image  flow  a practical  mechanism  for  enabling  high  speed  robot  mobility.  The 
paper  begins  by  describing  the  camera-centered  coordinate  systems  that  are  used  here, 
followed  by  deriving  expressions  that  relate  optical  flow  and  distance  of  a point  to  six- 
degree -of-freedom  camera  motion.  When  the  optical  flow  constraint  (as  described  by 
Horn  and  Schunck  [4])  is  added  to  these  equations,  a pixel-based  closed-form  solution 
for  the  3D  location  of  a point  in  the  camera  coordinate  system  is  obtained. 

2.  EQUATIONS  OF  MOTION  AND  OPTICAL  FLOW 

This  section  describes  the  equations  that  relate  a point  in  3-D  space  to  the  projec- 
tion of  that  point  in  the  image  for  general  six-degree -of- freedom  motion  of  the  camera. 
Some  of  the  equations  are  well  known  [11]. 

Assume  a moving  camera  in  a stationary  environment  with  the  coordinate  system 
fixed  with  respect  to  the  camera  as  shown  in  Figure  1.  Assume  a pinhole  camera 
model,  such  that  the  pinhole  of  the  camera  is  at  the  origin  of  the  coordinate  system. 
We  derive  the  optical  flow  components  in  the  spherical  coordinate  system  (/?9(1)).  In 
this  frame,  angular  yelocities  (9  and  (j))  of  any  point  in  space,  say  P,  are  identical  to  the 
optical  flow  values  9 and  (j)  in  the  image  domain.  See  Figure  2. 

The  relationship  between  the  image  flow  rates  in  the  spherical  domain  of  Figure  2 
and  the  velocity  of  a point  in  the  cartesian  coordinates  of  Figure  1 is  given  by  the  fol- 
lowing equations: 
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Let  the  instantaneous  cartesian  coordinates  of  the  point  P be  R = (X,r^)^ 
(where  the  superscript  T denotes  trangjose).  Let  the  instantaneous  translational  velo- 
city of  the  camera  be  t = {U  ,V  ,WY  and  the  instantaneous  angular  velocity  be 
. Then  the  velocity  vector  V of  the  point  P with  respect  to  the  XY2 
coordinate  system  is  (see  [11]): 


V = -t--©xR  (1) 

on 

Vx  = -U-BZ+CY  (2) 

Vj  ^ -V-CX+AZ  (3) 

= -W~AY+BX  (4) 


where  Vx,  Vy^  die  components  of  the  velocity  vector  V along  the  X,  Y , 

and  Z directions  respectively.  Let  Sq-  sin9,  Cq-  cos9,  5^  = sin^,  = cos(j), 
and  R - iRl.  To  convert  from  XYZ  to  R 94>  coordinates  we  use  the  relations: 


X ^R 

F =/?  C(j,  ^9 

Z = R 


(5) 

(6) 
(7) 


In  order  to  find  the  optical  flow  of  a 3-D  point  in  ^ 9(j)  coordinates,  we  use  the  follow- 
ing relations  and  transformations  (see  [12]  and  Figure  1).  Let  V^,  Vq,  and  be  the 
components  of  the  vector  V in  spherical  coordinates,  and 


C(J,  0 

Cq  Sq  0 

1 

II 

0 1 0 

-Sq  Cq  0 

-5<j)  0 

0 0 1 

1 

(8) 


Also  from  [12] 


II 

(9) 

Vq-RO  c^. 

(10) 

(11) 

where  dot  denotes  derivative  with  respect  to  time. 


Using  equations  (2)-(ll)  the  following  expressions  are  obtained: 


9 

1 

0 

CD 

05 

1 

R 0 

-U-B  R s^  + C R ^9 
-V~  C R Cq  + A R 
-W-  A R c^Sq  + B R Cq 


(12) 


There  are  three  unknowns  in  equation  set  (12):  R , 9,  and  (|).  For  each  pixel,  9 and  0 are 
known.  The  motion  parameters  (A  ^ ,C  ,U  y ,W)  are  also  known. 


3.  SOLUTIONS 

3.1  GRADIENT-BASED  METHOD 

3.1.1  THE  OPTICAL  FLOW  CONSTRAINT 


If  brightness  / varies  smoothly  with  0,  0,  and  t , then  we  obtain  the  optical  flow 
constraint  equation  (see  [4]): 


M. 

90 


(13) 


or 

+ + = 0 (14) 

where 

T — r _ 

90  ^^"90  " 9r’ 


I Q,  /(j,,  and  If  can  be  approximated  from  the  image  sequence  as  follows.  For  a given 
pixel  (0j-  ,0y ) (Figure  3)  in  the  image  at  time  instant 


9/  ^ /(0t,0;,r;fc)-/(0t-i,0y.rA:) 
90, • Qf-Qi-i 


9/  _ f (®t  >0y— l>^jk ) 

90y  0;- 0;-l 

9/  ^ /(0i,0y,r;fc)  -/(0,-,0y,f^.i) 


(16) 

(17) 


3.1.2  DEPTH  COMPUTATIONS 

Equation  set  (12)  together  with  equation  (14)  form  a set  of  three  equations  with 
three  unknowns  R , 0,  and  0.  Solving  these  equations  for  R yields: 

{USq-VCq)Iq+C^{Us^Cq+VS(^Sq-Wc^)I^ 

R — ■ (18) 

-c^f  - C^(5c0-A5e)/<J,  + (Cc^-^^(5^0+Ace))/9 

The  meaning  of  solution  (18)  is  that  if  the  location  of  the  pixel  in  the  image,  the 
motion  parameters  of  the  camera,  and  the  spatial  and  temporal  intensity  changes  are 
given,  then  the  depth  of  the  corresponding  point  in  3-D  can  be  obtained. 

At  first  inspection,  21  multiplications  and  divisions,  and  10  summations  and  sub- 
tractions appear  to  be  necessary  for  each  pixel  at  each  instant  of  time.  However,  for  a 
given  pixel,  0 and  0 are  constants  and  therefore  all  functions  of  0 and  0 such  as  CqC^ 
can  be  precomputed.  Also,  the  rate  of  change  of  the  motion  parameters 
A ,B  ,C  ,U  y ,W  is  usually  much  lower  than  the  rate  of  change  of  the  image  variables 
Iq,  and  /,.  Thus  the  update  rate  of  the  motion  parameters  can  be  significantly 
smaller  than  the  computation  of  R . For  example,  for  a translation  in  the  XY  plane,  and 
rotation  about  the  Z axis  (i.e..  A,  B,  and  W are  zero),  U,V,  and  C may  be  constants 
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for  some  period  of  time  but  Iq,  /,j,  and  /,  change.  Practically,  accelerations  (either  trans- 
lational or  rotational)  diQ  limited  due  to  mass  (or  inertia)  consttaints.  Thus,  any  func- 
tion of  ^ ,C  ,U ,V  W will  typically  have  an  update  rate  that  an  order  of  magni- 
tude smaller  than  the  intensity  changes  rate.  Equation  (18)  can  be  rewriten  as  follows: 

R = / ^ ^ ^ ^ ^ (19) 

a^It  +a4/^  + ^5/@ 

where  ai  “ Us^-Vc^  , a2  ~ c^(£/j^C0+V5(|,S9-WCe^)  , a3  = - , 

«4  - c^{Asq-Bcq)  , and  - Cc^^-s^iBsQ+AcQ)  . By  observing  equation  (19)  it  is 
clear  that  for  given  Iq^  and  /,,  at  the  fast  sampling  rate  there  are  only  five  multipli- 
cations, one  division,  and  three  summations  per  pixel. 

For  each  point  in  space  there  is  a set  of  three  equations  that  result  in  a reconstruc- 
tion for  R . The  expression  for  R that  corresponds  to  a pixel  in  the  image  can  be  com- 
puted using  a pixel-based  processor,  and  special  purpose  computers  may  run  in  parallel 
to  reconstruct  all  visible  points  in  space.  Figure  4 shows  a possible  multi-rate  structure. 
For  each  pixel  there  is  a "fast"  computer,  and  a "slow"  computer.  The  first  part  of  the 
"fast  computer"  extracts  Iq,  /^,  and  /^,  and  the  second  part  of  this  computer  computes 
the  distance  R that  corresponds  to  this  pixel.  The  latter  part  accepts  spatial  and  tem- 
poral intensity  changes  functions  as  well  as  motion  parameter  functions  for  this  particu- 
lar pixel  (i.e.,  ...  ,£15  in  equation  (19)),  and  combines  them  for  instantaneous  dis- 

tance exttaction.  The  "slow"  computer  generates  functions  of  -B  ,C  ,U  ,V  ,W  at  a 
low  update  rate. 

It  is  difficult  to  analyze  the  error  since  there  are  eight  parameters 
B ,C  ,U  ,V  ,W)  to  consider.  However,  there  are  singular  points,  i.e.,  combinations 
of  the  parameters  and  variables  values  that  cause  the  denominator  to  be  zero.  There 
are  also  points  that  cannot  be  reconstructed,  e.g.,  those  that  lie  exactly  on  the  direction 
of  the  motion  vector. 

3.1.3  A SPECIAL  CASE 


In  order  to  inmitively  understand  expression  (18),  we  show  a special  case.  Sup- 
pose that  the  camera  undergoes  nanslation  U and  V in  the  instantaneous  XY  plane  and 
rotation  C about  the  Z axis  only,  i.e.,  A - B - W = 0.  Assume  (|)  = 0,  i.e.,  the  pixel 
which  is  analyzed  lies  on  the  XY  plane.  Substituting  these  values  in  (18)  results  in  the 
following  expression  for  R : 


(Usq-Vcq)I  Q 

-/r  +C/9 


(20) 


or  (assuming  Iqj^) 


Usq  - Vcq 
-e  + C 


(21) 


where  0=“ 
/e 


Given  that  , equation  (21)  is  a solution  for  any  point  in  the  XY  plane 
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when  translating  in  this  plane  and  rotating  about  the  Z axis. 
3.2  CORRELATION  METHOD 


Computation  of  image  flow  by  cross-correlation  is  an  alternative  to  using  the  opti- 
cal flow  constraint  equation  (13).  Cross  correlation  also  requires  that  the  direction  of 
the  flow  vector  be  known.  Assume  that  for  any  point  in  space  viewed  by  a pixel  at  an 
instant  of  tme,  the  direction  of  the  image  flow  vector  can  be  found,  i.e.,  assume  that 
0 

the  ratio  — is  known.  A second  pixel  can  then  be  selected  at  some  angular  distance 
(|> 

A4  from  the  first  along  the  flow  direction.  One  can  then  compute  a cross-correlation 
function  between  the  first  and  second  pixels.  This  will  peak  at  some  delay  interval 

dA 

and  the  image  flow  rate  — of  the  point  in  space  is  then  given  by  (see  [13]): 

at 


dA  AA 

- — = lim  — — 
dt 


(22) 


It  can  be  approximated  by: 
dA  _ M 
dt 


(23) 


“max 


After  computing  and  the  ratio  the  values  of  (j)  and  9 can  be  computed  from: 
dt  (|) 

1 dA  . -1 9 X 

0 = cos  (tan  ^-r-) 

dt 


A dA  . . -10. 

9 = — — sm  (tan  ^— ) 
dt 


<1> 

(j) 


(24) 

(25) 


The  value  of  R can  be  computed  from  equation  set  (12). 

0 

For  a general  motion  -r-  is  unknown.  Thus  several  pixels  in  the  neighborhood  of 

the  first  one  has.  to  be  selected.  The  pixel  for  which  the  cross  correlation  function 
0 

peaks  provides  — , AA , and  x^gx’  same  (previously  described)  procedure  can 

be  used  to  compute  R . 

4.  DISCUSSION 

Computing  range  from  image  flow  by  the  method  of  Horn  and  Schunck  [4]  has 
the  advantage  of  simplicity.  Unfonunateiy,  it  tends  to  produce  noisy  and  inaccurate 
results.  There  are  four  sources  of  noise  and  errors: 

First,  the  sampled  data  system  approximations  to  and  are  subject  to 

digitization  noise  and  Nyquist  sampling  frequency  limitations.  If  /(9,0,r)  contains  spa- 
tial (or  temporal)  frequency  components  that  are  higher  than  half  the  spatial  (or  tem- 
poral) sampling  frequency,  aliasing  may  occur.  Of  course,  the  natural  world  is  infinitely 
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rich  in  detml.  Thus  low-pass  filtering  must  be  perfonned  on  natural  images  to  smooth 
them  before  the  Horn  and  Schunck  method  can  be  used.  The  resultant  blurring  of  the 
images  removes  sharp  features  that  contain  the  most  accurate  information  about  image 
flow. 

Second,  except  in  the  vicinity  of  brighmess  edges,  ■”  is  small,  and  may  be  zero. 
Division  by  small  numbers  magnifies  errors. 


Third,  the  sensitivity  of  photodetectors  in  any  array  is  not  unifonn.  The  difference 
in  signal  from  two  adjacent  photo-detectors  thus  is  not  necessarily  an  accurate  measure 
of  the  difference  in  illumination.  In  fact,  in  the  biologicai  eye,  where  accomodation 
occurs  in  the  retinal  photodetectors,  if  the  eye  is  fixated  for  an  extended  period,  accom- 
modation causes  the  image  to  fade  away  completely.  In  this  case,  difference  in  output 
of  adjacent  photodetectors  provides  no  information  at  aU  about  the  spatial  brighmess 
gradient 

Fourth,  thermal  noise  in  photodetectors  is  not  "white",  but  heavily  biased  toward 
low  frequencies.  The  difference  in  signal  from  two  adjacent  photodetectors  is  thus  sub- 
ject to  large  amounts  of  differential  low  frequency  noise.  Thus  range  from  image  flow 
by  the  Horn  and  Schunck  method  tends  to  be  inaccurate  for  smooth  images,  and  unreli- 
able for  sharply  focused  natural  images. 

Computing  range  from  image  flow  by  the  method  of  cross-correlation  can  be 
much  more  accurate  and  noise  free.  It  is,  however,  more  complicated  and  computation- 
ally intensive.  The  selection  of  which  two  pixels  to  correlate  is  a function  of  the  trans- 
lational and  rotational  velocities.  For  each  pixel  where  correlation  is  performed,  a 
delay  line,  a cross  correlator,  and  a correlation  peak  detector  are  required  [13].  The 
accuracy  is  limited  by  the  ratio  of  the  sampling  rate  to  the  delay  For  one  percent 
accuracy,  the  intersample  interval  must  be  100  times  less  than  tjuax- 


In  order  to  reduce  noise  and  increase  accuracy,  temporal  integration  methods  can 
be  used.  Methods  for  performing  temporal  integration  include  Kalman  filtering 
[9], [10],  and  computing  the  mnning  average  in  world  coordinates  [13].  Temporal 
integration  by  running  averaging  has  been  shown  to  reduce  inaccuracies  in  the  compu- 
tation of  depth  from  image  flow  by  the  Horn  and  Schunck  method  by  about  an.  order  of 
magnitude,  from  about  30%  absolute  depth  error  in  the  raw  data  to  about  3%  absolute 
depth  error  in  the  averaged  data  [13].  This  is  comparable  to  the  results  achieved  by  the 
much  more  computationally  intensive  cross-correlation  and  Kalman  filtering  methods  of 
Matties  et  ai  [10].  Recent  results  achieved  by  a new  correlation-integration  method 
[14]  show  absolute  depth  errors  of  less  than  0.6%. 

5.  CONCLUSIONS 


When  eye  velocity  is  known,  image  flow  can  be  a simple,  robust,  and  powerful 
method  for  generating  dense  range  images.  For  complex  natural  scenes,  dense  range 
images  computed  from  image  flow  should  prove  to  be  much  easier  to  segment  that 
brightness  images.  The  combination  of  image  flow  techniques  with  more  traditional 
methods  thus  promises  to  vastly  improve  the  performance  of  machine  vision  systems. 
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The  ability  to  compute  dense  range  images  without  knowledge  of,  or  information 
about,  the  contents  of  scenes  may  explain  the  ability  of  creatures  of  nature,  such  as 
flying  insects,  birds,  fish,  and  mammals,  to  manuever  in  a complex  natural  world. 
Accuracy  of  range  fi"om  image  flow  is  proportional  to  accuracy  of  velocity.  For  many 
important  robotics  applications,  such  as  driving  unmanned  ground  vehicles,  velocity  is 
known  well  enough  to  compute  useful  range  from  image  flow.  Image  flow  may  thus 
become  a practical  method  for  obstacle  avoidance  in  unmanned  vehicles. 

When  eye  rotation  is  zero,  the  computation  of  image  flow,  and  range  from  image 
flow,  is  quite  simple.  This  may  explain  why  many  creatures  of  nature  have  a vestibular 
reflex  that  can  mechanically  stabilize  the  eye  against  rotation.  There  is  also  psychophy- 
sical evidence  for  neuronal  mechanisms  that  scroU  the  retinal  image  so  as  to  cause  the 
visual  world  to  be  subjectively  perceived  as  stationary  despite  eye  rotation  that  causes 
the  retinal  image  to  move  about 

This  paper  has  shown  how  depth  from  image  flow  can  be  computed  directly  from 
local  image  parameters,  given  knowledge  of  eye  translation  and  rotaion.  The  closed 
form  algorithm  that  has  been  presented  is  amenable  to  implementation  on  massively- 
parallel  single-instruction  multiple  data  (SIMD)  computing  machines.  It  could,  in  fact, 
be  computed  by  a single  layer  of  neurons  of  the  type  known  to  exist  in  the  biological 
brain. 

Future  work  will  focus  on  measures  of  performance  of  the  above  algorithms  in  an 
environment  filled  with  a variety  of  natural  objects  such  as  trees,  bushes,  rocks, 
streams,  dirt,  leaves,  and  grass.  Tests  will  be  made  that  simulate  tasks  such  as  driving, 
running,  or  flying  at  high  speeds,  like  insects  and  birds  routinely  do,  through  complex 
natural  environments  such  as  woods  and  fields. 
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